Papers with Contextual word representations
Knowledge Enhanced Contextual Word Representations (D19-1)
Copied to clipboard
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, Noah A. Smith
| Challenge: | Existing methods to embed knowledge bases into large pre-training models do not contain any explicit grounding to real world entities and are difficult to recover factual knowledge. |
| Approach: | They propose a method to embed multiple knowledge bases (KBs) into large pretrained models with a Knowledge Attention and Recontextualization mechanism. |
| Outcome: | The proposed model improves perplexity, ability to recall facts and word sense disambiguation. |
Linguistic Knowledge and Transferability of Contextual Representations (N19-1)
Copied to clipboard
| Challenge: | Recent work has explored contextual word representations, which assign each word a vector that is a function of the entire input sequence. |
| Approach: | They compare pretrained word representations with 16 diverse probing tasks to examine their transferability. |
| Outcome: | The pretrained representations are successful across a diverse set of NLP tasks . the models are competitive with state-of-the-art models but fail on fine-grained tasks requiring fine-granular knowledge, the study finds . |
Dissecting Contextual Word Embeddings: Architecture and Representation (D18-1)
Copied to clipboard
| Challenge: | Existing work on learning contextual representations has used LSTM-based biLMs, but there is no reason to believe this is effective. |
| Approach: | They propose to use pre-trained bidirectional language models to learn contextual word embeddings for four NLP tasks and to use them to study the effects of architecture on endtask accuracy. |
| Outcome: | The proposed models outperform word embeddings for four NLP tasks and all learn representations that vary with network depth. |